from sklearn_benchmarks.reporting.hp_match import HpMatchReporting

scikit-learn vs. ONNX Runtime

reporting = HpMatchReporting(other_library="onnx", config="config.yml", log_scale=True)
reporting.make_report()

We assume here there is a perfect match between the hyperparameters of both librairies. For a given set of parameters and a given dataset, we compute the speedup time scikit-learn / time onnx. For instance, a speedup of 2 means that onnx is twice as fast as scikit-learn for a given set of parameters and a given dataset.

KNeighborsClassifier_brute_force

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: algorithm=brute.

WARNING! Mismatch between validation scores for 12 prediction (75.0%). The observed differences can be found in the diff_accuracy_scores column. The chosen difference threshold is 0.001 for accuracy_score. See details in the dataframe below.

	estimator	function	n_samples_train	n_samples	n_features	parameters_digest	dataset_digest	mean_duration_sklearn	std_duration_sklearn	n_iter	...	algorithm	n_jobs	n_neighbors	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	diff_accuracy_scores
1	KNeighborsClassifier_brute_force	predict	100000	1000	100	e70d8e7886766d41f30889506baa1e67	2cf49829391aa7891e877f2ed070adf0	1.863516	0.117614	NaN	...	brute	-1	1	0.663	0.332630	0.020317	1.000	5.602372	5.612813	0.337
2	KNeighborsClassifier_brute_force	predict	100000	1	100	e70d8e7886766d41f30889506baa1e67	2cf49829391aa7891e877f2ed070adf0	0.022580	0.001318	NaN	...	brute	-1	1	1.000	17.518909	0.020043	0.757	0.001289	0.001289	0.243
4	KNeighborsClassifier_brute_force	predict	100000	1000	100	22d5eac4a4146149b35eae6914921514	2cf49829391aa7891e877f2ed070adf0	2.727397	0.029766	NaN	...	brute	-1	5	0.757	17.665339	0.219245	0.882	0.154393	0.154404	0.125
7	KNeighborsClassifier_brute_force	predict	100000	1000	100	16f94fc93942ff7ece860c9b0f64645f	2cf49829391aa7891e877f2ed070adf0	2.036001	0.034826	NaN	...	brute	1	100	0.882	0.302218	0.010304	1.000	6.736870	6.740785	0.118
8	KNeighborsClassifier_brute_force	predict	100000	1	100	16f94fc93942ff7ece860c9b0f64645f	2cf49829391aa7891e877f2ed070adf0	0.019218	0.000417	NaN	...	brute	1	100	1.000	17.958458	0.024560	0.757	0.001070	0.001070	0.243
10	KNeighborsClassifier_brute_force	predict	100000	1000	100	2d5f2b2c02d77766434d9e5033b4a76c	2cf49829391aa7891e877f2ed070adf0	2.735226	0.044376	NaN	...	brute	-1	100	0.882	17.927651	0.072061	0.663	0.152570	0.152571	0.219
13	KNeighborsClassifier_brute_force	predict	100000	1000	100	efac583623711d395ddae7a1e881ff68	2cf49829391aa7891e877f2ed070adf0	2.058051	0.054348	NaN	...	brute	1	5	0.757	0.254465	0.014530	1.000	8.087749	8.100923	0.243
14	KNeighborsClassifier_brute_force	predict	100000	1	100	efac583623711d395ddae7a1e881ff68	2cf49829391aa7891e877f2ed070adf0	0.019395	0.000586	NaN	...	brute	1	5	1.000	3.730376	0.082546	0.922	0.005199	0.005201	0.078
16	KNeighborsClassifier_brute_force	predict	100000	1000	100	b79d63bb7670c492de5c3befac58fe29	2cf49829391aa7891e877f2ed070adf0	1.134952	0.023934	NaN	...	brute	1	1	0.663	3.733879	0.059970	0.929	0.303961	0.304000	0.266
19	KNeighborsClassifier_brute_force	predict	100000	1000	2	e70d8e7886766d41f30889506baa1e67	88843f54689e3271092f70126e1de585	1.678147	0.028749	NaN	...	brute	-1	1	0.896	0.279361	0.007953	1.000	6.007099	6.009533	0.104
20	KNeighborsClassifier_brute_force	predict	100000	1	2	e70d8e7886766d41f30889506baa1e67	88843f54689e3271092f70126e1de585	0.004615	0.001554	NaN	...	brute	-1	1	1.000	4.157028	0.032592	0.922	0.001110	0.001110	0.078
22	KNeighborsClassifier_brute_force	predict	100000	1000	2	22d5eac4a4146149b35eae6914921514	88843f54689e3271092f70126e1de585	2.584159	0.029533	NaN	...	brute	-1	5	0.922	4.166782	0.051759	0.896	0.620181	0.620229	0.026

12 rows × 22 columns

Raw results

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.011	0.000	7.295	0.0	-1	1	17.823	0.537	0.663	0.001	0.001	See	See
3	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.011	0.000	7.101	0.0	-1	5	0.325	0.014	1.000	0.035	0.035	See	See
6	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.011	0.000	7.214	0.0	1	100	17.350	0.290	0.882	0.001	0.001	See	See
9	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.011	0.000	7.269	0.0	-1	100	0.330	0.013	1.000	0.033	0.033	See	See
12	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.011	0.001	7.325	0.0	1	5	3.780	0.086	0.896	0.003	0.003	See	See
15	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.011	0.000	7.236	0.0	1	1	0.237	0.008	1.000	0.047	0.047	See	See
18	KNeighborsClassifier_brute_force	fit	100000	100000	2	0.005	0.000	0.346	0.0	-1	1	4.042	0.161	0.929	0.001	0.001	See	See
21	KNeighborsClassifier_brute_force	fit	100000	100000	2	0.005	0.000	0.332	0.0	-1	5	0.281	0.006	1.000	0.017	0.017	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	KNeighborsClassifier_brute_force	predict	100000	1000	100	1.864	0.118	0.000	0.002	-1	1	0.333	0.020	1.000	5.602	5.613	See	See
2	KNeighborsClassifier_brute_force	predict	100000	1	100	0.023	0.001	0.000	0.023	-1	1	17.519	0.020	0.757	0.001	0.001	See	See
4	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.727	0.030	0.000	0.003	-1	5	17.665	0.219	0.882	0.154	0.154	See	See
5	KNeighborsClassifier_brute_force	predict	100000	1	100	0.022	0.002	0.000	0.022	-1	5	0.319	0.012	1.000	0.070	0.070	See	See
7	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.036	0.035	0.000	0.002	1	100	0.302	0.010	1.000	6.737	6.741	See	See
8	KNeighborsClassifier_brute_force	predict	100000	1	100	0.019	0.000	0.000	0.019	1	100	17.958	0.025	0.757	0.001	0.001	See	See
10	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.735	0.044	0.000	0.003	-1	100	17.928	0.072	0.663	0.153	0.153	See	See
11	KNeighborsClassifier_brute_force	predict	100000	1	100	0.023	0.002	0.000	0.023	-1	100	0.315	0.011	1.000	0.073	0.073	See	See
13	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.058	0.054	0.000	0.002	1	5	0.254	0.015	1.000	8.088	8.101	See	See
14	KNeighborsClassifier_brute_force	predict	100000	1	100	0.019	0.001	0.000	0.019	1	5	3.730	0.083	0.922	0.005	0.005	See	See
16	KNeighborsClassifier_brute_force	predict	100000	1000	100	1.135	0.024	0.001	0.001	1	1	3.734	0.060	0.929	0.304	0.304	See	See
17	KNeighborsClassifier_brute_force	predict	100000	1	100	0.019	0.001	0.000	0.019	1	1	0.247	0.012	1.000	0.078	0.078	See	See
19	KNeighborsClassifier_brute_force	predict	100000	1000	2	1.678	0.029	0.000	0.002	-1	1	0.279	0.008	1.000	6.007	6.010	See	See
20	KNeighborsClassifier_brute_force	predict	100000	1	2	0.005	0.002	0.000	0.005	-1	1	4.157	0.033	0.922	0.001	0.001	See	See
22	KNeighborsClassifier_brute_force	predict	100000	1000	2	2.584	0.030	0.000	0.003	-1	5	4.167	0.052	0.896	0.620	0.620	See	See
23	KNeighborsClassifier_brute_force	predict	100000	1	2	0.010	0.005	0.000	0.010	-1	5	0.275	0.005	1.000	0.038	0.038	See	See

KNeighborsClassifier_kd_tree

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: algorithm=kd_tree.

WARNING! Mismatch between validation scores for 12 prediction (75.0%). The observed differences can be found in the diff_accuracy_scores column. The chosen difference threshold is 0.001 for accuracy_score. See details in the dataframe below.

	estimator	function	n_samples_train	n_samples	n_features	parameters_digest	dataset_digest	mean_duration_sklearn	std_duration_sklearn	n_iter	...	algorithm	n_jobs	n_neighbors	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	diff_accuracy_scores
1	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	fe8267e25dadb302ed498e50fabf7ef4	1eb6cf1a720a225efb91df0b529b0510	0.826547	1.011824	NaN	...	kd_tree	-1	1	0.929	2.915676	0.212761	1.000	0.283484	0.284238	0.071
2	KNeighborsClassifier_kd_tree	predict	1000000	1	10	fe8267e25dadb302ed498e50fabf7ef4	1eb6cf1a720a225efb91df0b529b0510	0.002777	0.000633	NaN	...	kd_tree	-1	1	1.000	118.027272	0.000000	0.946	0.000024	0.000024	0.054
4	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	dc74b969bf622bc24ba3bc62c980983b	1eb6cf1a720a225efb91df0b529b0510	1.012278	0.332108	NaN	...	kd_tree	-1	5	0.946	118.385681	0.000000	0.951	0.008551	0.008551	0.005
7	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	41accfcdcd5c50784bedf6164b63de99	1eb6cf1a720a225efb91df0b529b0510	5.421563	0.349170	NaN	...	kd_tree	1	100	0.951	2.968757	0.133299	1.000	1.826206	1.828046	0.049
8	KNeighborsClassifier_kd_tree	predict	1000000	1	10	41accfcdcd5c50784bedf6164b63de99	1eb6cf1a720a225efb91df0b529b0510	0.003874	0.000806	NaN	...	kd_tree	1	100	1.000	117.224741	0.000000	0.946	0.000033	0.000033	0.054
10	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	022f7445d43bb1dbc24dc3106c03cb93	1eb6cf1a720a225efb91df0b529b0510	3.141040	0.143311	NaN	...	kd_tree	-1	100	0.951	116.579537	0.000000	0.929	0.026943	0.026943	0.022
13	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	84622ba45e941db642965553529e1941	1eb6cf1a720a225efb91df0b529b0510	1.625557	0.200726	NaN	...	kd_tree	1	5	0.946	0.005819	0.000324	1.000	279.358887	279.792285	0.054
14	KNeighborsClassifier_kd_tree	predict	1000000	1	10	84622ba45e941db642965553529e1941	1eb6cf1a720a225efb91df0b529b0510	0.001688	0.000513	NaN	...	kd_tree	1	5	1.000	0.043705	0.001130	0.911	0.038620	0.038633	0.089
16	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	a9455565db1d8e052a783317c99744ff	1eb6cf1a720a225efb91df0b529b0510	0.859269	0.173876	NaN	...	kd_tree	1	1	0.929	0.068548	0.006555	0.894	12.535376	12.592562	0.035
19	KNeighborsClassifier_kd_tree	predict	1000	1000	2	fe8267e25dadb302ed498e50fabf7ef4	edf7cb014c2c0910e1c3aaaf8ae633f2	0.027149	0.012476	NaN	...	kd_tree	-1	1	0.891	0.005732	0.000102	1.000	4.736063	4.736806	0.109
20	KNeighborsClassifier_kd_tree	predict	1000	1	2	fe8267e25dadb302ed498e50fabf7ef4	edf7cb014c2c0910e1c3aaaf8ae633f2	0.002387	0.000322	NaN	...	kd_tree	-1	1	1.000	0.042454	0.001367	0.911	0.056231	0.056260	0.089
22	KNeighborsClassifier_kd_tree	predict	1000	1000	2	dc74b969bf622bc24ba3bc62c980983b	edf7cb014c2c0910e1c3aaaf8ae633f2	0.024406	0.001728	NaN	...	kd_tree	-1	5	0.911	0.042142	0.000999	0.891	0.579137	0.579299	0.020

12 rows × 22 columns

Raw results

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	2.986	0.098	0.027	0.0	-1	1	119.164	0.000	0.929	0.025	0.025	See	See
3	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.624	0.073	0.022	0.0	-1	5	2.885	0.179	1.000	1.256	1.259	See	See
6	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.715	0.080	0.022	0.0	1	100	118.157	0.000	0.951	0.031	0.031	See	See
9	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.796	0.114	0.021	0.0	-1	100	2.948	0.113	1.000	1.287	1.288	See	See
12	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.780	0.060	0.021	0.0	1	5	0.056	0.016	0.891	67.675	70.526	See	See
15	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.727	0.104	0.021	0.0	1	1	0.006	0.000	1.000	671.177	671.564	See	See
18	KNeighborsClassifier_kd_tree	fit	1000	1000	2	0.001	0.001	0.021	0.0	-1	1	0.067	0.001	0.894	0.011	0.011	See	See
21	KNeighborsClassifier_kd_tree	fit	1000	1000	2	0.001	0.000	0.027	0.0	-1	5	0.006	0.000	1.000	0.100	0.100	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.827	1.012	0.000	0.001	-1	1	2.916	0.213	1.000	0.283	0.284	See	See
2	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.001	0.000	0.003	-1	1	118.027	0.000	0.946	0.000	0.000	See	See
4	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	1.012	0.332	0.000	0.001	-1	5	118.386	0.000	0.951	0.009	0.009	See	See
5	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.001	0.000	0.003	-1	5	2.993	0.195	1.000	0.001	0.001	See	See
7	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	5.422	0.349	0.000	0.005	1	100	2.969	0.133	1.000	1.826	1.828	See	See
8	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.004	0.001	0.000	0.004	1	100	117.225	0.000	0.946	0.000	0.000	See	See
10	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	3.141	0.143	0.000	0.003	-1	100	116.580	0.000	0.929	0.027	0.027	See	See
11	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.005	0.001	0.000	0.005	-1	100	2.942	0.152	1.000	0.002	0.002	See	See
13	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	1.626	0.201	0.000	0.002	1	5	0.006	0.000	1.000	279.359	279.792	See	See
14	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.002	0.001	0.000	0.002	1	5	0.044	0.001	0.911	0.039	0.039	See	See
16	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.859	0.174	0.000	0.001	1	1	0.069	0.007	0.894	12.535	12.593	See	See
17	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.001	0.000	0.000	0.001	1	1	0.006	0.000	1.000	0.185	0.185	See	See
19	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.027	0.012	0.001	0.000	-1	1	0.006	0.000	1.000	4.736	4.737	See	See
20	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.002	0.000	0.000	0.002	-1	1	0.042	0.001	0.911	0.056	0.056	See	See
22	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.024	0.002	0.001	0.000	-1	5	0.042	0.001	0.891	0.579	0.579	See	See
23	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.002	0.000	0.000	0.002	-1	5	0.005	0.000	1.000	0.389	0.389	See	See

HistGradientBoostingClassifier_best

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: learning_rate=0.01, n_iter_no_change=10.0, max_leaf_nodes=100.0, max_bins=255.0, min_samples_leaf=100.0, max_iter=300.0.

WARNING! Mismatch between validation scores for 1 prediction (100.0%). The observed differences can be found in the diff_accuracy_scores column. The chosen difference threshold is 0.001 for accuracy_score. See details in the dataframe below.

	estimator	function	n_samples_train	n_samples	n_features	parameters_digest	dataset_digest	mean_duration_sklearn	std_duration_sklearn	n_iter	...	max_leaf_nodes	min_samples_leaf	n_iter_no_change	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	diff_accuracy_scores
1	HistGradientBoostingClassifier_best	predict	100000	1000	100	25b9f14bed7dd5d830c6ccd4dfebbf0c	8c8fffa8ec4b2d8de833421f9e32beab	0.120581	0.002811	300	...	100	100	10	0.824	0.430468	0.009203	1.0	0.280115	0.280179	0.176

1 rows × 25 columns

Raw results

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	HistGradientBoostingClassifier_best	fit	100000	100000	100	100.029	0.0	300	0.001	0.001	0.539	0.027	0.824	185.724	185.959	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	HistGradientBoostingClassifier_best	predict	100000	1000	100	0.121	0.003	300	0.007	0.0	0.43	0.009	1.0	0.28	0.28	See	See